Skip to content

Add Population Threshold RDD Italy dataset files - #18

Merged
ankurankan merged 1 commit into
pgmpy:mainfrom
nicolasdmolina:feature/population-threshold-rdd-italy
Sep 16, 2026
Merged

ankurankan merged 1 commit into
pgmpy:mainfrom
nicolasdmolina:feature/population-threshold-rdd-italy

Conversation

@nicolasdmolina

Copy link
Copy Markdown
Contributor

Adds the Italian municipality-year table from Eggers et al. (AJPS 2018) so pgmpy can load population_threshold_rdd_italy.

Source: Harvard Dataverse doi:10.7910/DVN/PGXO5O
License: CC0 1.0
Original zip MD5: df3c2bd36a9f5fcd9d1e86d5b8e31f66
Packaged file SHA256: 608e33e1907f573e991d206ea72827b7b65c674f514d3a5f37ae754ba1430846

This is the data companion for pgmpy PR pgmpy/pgmpy#3566 (issue #2788 / parent #2620).

CC0 replication extract from Eggers et al. AJPS 2018 (Harvard Dataverse
doi:10.7910/DVN/PGXO5O), with license, source, and checksum docs.
@nicolasdmolina

Copy link
Copy Markdown
Contributor Author

Bumping for visibility. Happy to make changes if there's anything blocking this.

@ankurankan ankurankan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay. Looks good.

@ankurankan
ankurankan merged commit a323a5c into pgmpy:main Sep 16, 2026
Mahajan-Sachin pushed a commit to Mahajan-Sachin/pgmpy that referenced this pull request Sep 16, 2026
**The following checklist is mandatory**

Your PR will be closed if you remove the checklist. Use of LLMs is
**strictly forbidden** for any part of this checklist (including for
improving language), and will result in a **ban** if we find any use of
LLMs.

### Your checklist for this pull request

- [x] Have you followed all the steps from our [Contributing
Guide](https://github.com/pgmpy/pgmpy/blob/dev/CONTRIBUTING.md)?
- [x] Does the PR fully address the linked issue and is within its
defined scope? If you are still working on the PR, mark it as draft.
- [x] Are all the GitHub Actions checks passing? If not, mark your PR as
draft while you fix it.

If you have used AI/LLMs for any assistance, please answer the following
questions. Please refer
[pgmpy#2622](pgmpy#2622) for an example of the
level of detail we expect:

- [x] Have you reviewed our [AI usage
policy](https://github.com/pgmpy/pgmpy/blob/dev/CONTRIBUTING.md#ai-usage-policy)?
- [x] Are you able to fully explain your changes? We expect you to fully
understand the algorithm and take full responsibility for any changes in
this PR.

- Please list the AI tool(s) used, along with the model and its version
used.

Grok 4.6 (xAI), used from the Grok Build TUI.

- Please describe in **detail** how and what you used AI assistance for?
Please outline your whole workflow with the AI tool.

I used Grok to search pgmpy#2620, related issues, and open PRs so I would not
take a dataset that already had an active PR. After I picked the Italy
table from the Eggers population-threshold RDD archive, I used it to
draft the BaseDataset subclass from
devtools/extension_templates/_dataset.py and the existing AngristKrueger
loader. I also used it to convert pop_devs_three_countries_v5.RData into
the tab-separated file, write the tag test, and add the two bibliography
entries. I read the resulting class, tags, and test myself before
opening this PR. I did not have it invent extra features or extra
datasets.

- What steps have you taken to verify that the changes correctly address
the issue? What edge cases have you considered? Other than running
tests, what else have you verified?

The Dataverse zip MD5 matched df3c2bd36a9f5fcd9d1e86d5b8e31f66. The
packaged Italy file is 10409 rows by 19 columns. I loaded that file
through PopulationThresholdRDDItaly.load_dataframe() locally (same
parser as the hub path) and checked n_samples / n_variables, missing
values, and macro_area as a category. I ran pytest -v
pgmpy/tests/test_datasets/test_datasets.py (10 passed) and pre-commit
ruff/format on the Python files. I did not add a live hub download test,
because the file is not on the Hub until the companion example_datasets
PR is merged. I left France and Germany out of this PR on purpose.

- Have you used AI for generating tests? Can you compress them into a
smaller number of tests without losing coverage?

Yes, for one tag/registry test. It only checks discovery, tag values,
the categorical column, and the missing-value marker. I did not add a
large generated suite.

### Issue number(s) that this pull request fixes
- Fixes pgmpy#2788
- Parent: pgmpy#2620 (this PR adds only the Population Threshold RDD Italy
table)

### List of changes to the codebase in this pull request
- Added PopulationThresholdRDDItaly in
pgmpy/datasets/population_threshold_rdd.py using the dataset extension
template (on-demand hub download + cache, metadata tags,
source/license/checksum in the docstring).
- Registered population_threshold_rdd_italy in ALL_DATASETS and added a
tag test.
- Added bibliography entries for the AJPS paper and the Harvard
Dataverse archive.

Hub data files (CC0, checksums in the README):
pgmpy/example_datasets#18
@nicolasdmolina
nicolasdmolina deleted the feature/population-threshold-rdd-italy branch September 26, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants